2026-03-05
toy example: Sections 18 and 19Connors et al. (1996) used a logistic regression model to develop a propensity score, then:
Key Conclusions: RHC patients had decreased survival time, and any unmeasured confounder would have to be somewhat strong to explain away the results.
Panel (7 specialists in clinical care) specified important variables related to the decision to use or not use a RHC.
This is part of the general class of sensitivity analyses but we also have a specific formal idea in mind for those analyses, when we have matched samples.
We want to assess the potential for unmeasured covariates to change our conclusions.
An omitted variable is most likely to change our conclusions about the exposure if it is
No.
| Corr. with PS | Covariates | Balance Improved | Median Bias Reduction |
|---|---|---|---|
| Sig. (p < .05) | 10 | 9 (90%) | 45% |
| Not Sig. | 7 | 2 (29%) | -36% |
Coronary bypass surgery or medical/drug therapy for CAD1?
| Substantial Improvement | Prob (SE) |
|---|---|
| Medical | 0.359 (0.042) |
| Surgical | 0.669 (0.059) |
Conclusion: Pr(improved | Surgery) > Pr(improved | medicine)
Conclusion: Pr(improved | Surgery) > Pr(improved | medicine)
Replace a qualitative and general statement that applies to all observational studies
with a quantitative statement specific to this study
How would inferences about treatment effects be altered by hidden biases of various magnitudes?
\(\Gamma\) measures degree of departure from a study that is free of hidden bias. A sensitivity analysis will consider possible values of \(\Gamma\) and show how the inference might change.
“To attribute the (observed significant) outcome to an unobserved covariate rather than to the treatment, that unobserved covariate has to increase the odds of treatment by a factor of \(\Gamma\), and also predict our outcome quite well.”
# A tibble: 500 × 7
subject treatment propensity out_binary out_quant censored out_time
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 S003 0 0.431 0 450. 1 99
2 S004 0 0.879 0 244. 0 585
3 S012 0 0.223 0 377. 0 268
4 S013 0 0.887 0 442. 0 798
5 S018 0 0.562 0 306. 0 687
6 S020 0 0.529 0 231. 0 108
7 S022 0 0.578 0 172. 0 212
8 S025 0 0.276 0 339. 0 672
9 S028 0 0.533 0 255. 0 590
10 S029 0 0.668 0 314. 0 441
# ℹ 490 more rows
sim_obs data subject treatment propensity out_binary
Length:500 Min. :0.0 Min. :0.0030 Min. :0.00
Class :character 1st Qu.:0.0 1st Qu.:0.2875 1st Qu.:0.00
Mode :character Median :0.0 Median :0.4530 Median :1.00
Mean :0.4 Mean :0.4592 Mean :0.59
3rd Qu.:1.0 3rd Qu.:0.6552 3rd Qu.:1.00
Max. :1.0 Max. :0.9380 Max. :1.00
out_quant censored out_time
Min. : 2.9 Min. :0.000 Min. : 17.0
1st Qu.:137.2 1st Qu.:0.000 1st Qu.: 283.8
Median :209.0 Median :0.000 Median : 543.5
Mean :217.4 Mean :0.172 Mean : 547.9
3rd Qu.:286.2 3rd Qu.:0.000 3rd Qu.: 808.5
Max. :466.7 Max. :1.000 Max. :1140.0
Why set a seed? Because if you don’t, the match can change on you (tied propensity scores…)
binarysens
Rosenbaum Sensitivity Test
Unconfounded estimate .... 0
Gamma Lower bound Upper bound
1.00 1e-05 0.00001
1.25 0e+00 0.00046
1.50 0e+00 0.00704
1.75 0e+00 0.04100
2.00 0e+00 0.12960
Note: Gamma is Odds of Differential Assignment To
Treatment Due to Unobserved Factors
binarysens matches subject treatment propensity out_binary out_quant censored out_time
1 301 S091 0 0.516 1 145.8 0 572
2 301 S001 1 0.516 0 260.0 1 735
3 302 S053 0 0.488 0 178.1 0 144
4 302 S005 1 0.487 0 334.7 0 149
5 303 S040 0 0.047 1 167.9 0 559
6 303 S027 1 0.044 0 225.8 0 994
| 2x2 Table | Treated has out1 |
Treated no out1 |
|---|---|---|
Control has out1 |
74 | 27 |
Control no out1 |
68 | 31 |
We have 200 matched pairs, and 95 pairs in the off-diagonal. There are 68 pairs where only the treated subject has the outcome. Assuming no hidden bias, a 95% CI for the McNemar odds ratio (68/27 or 2.52) is…
Estimate... -56.681
SE......... 9.99
T-stat..... -5.6738
p.val...... 1.397e-08
Original number of observations.............. 500
Original number of treated obs............... 200
Matched number of observations............... 200
Matched number of observations (unweighted). 200
psens to estimate \(\Gamma\)
Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value
Unconfounded estimate .... 0
Gamma Lower bound Upper bound
1.00 0 0.0000
1.25 0 0.0001
1.50 0 0.0036
1.75 0 0.0353
2.00 0 0.1453
2.25 0 0.3453
2.50 0 0.5753
Note: Gamma is Odds of Differential Assignment To
Treatment Due to Unobserved Factors
Rosenbaum Sensitivity Test for Hodges-Lehmann Point Estimate
Unconfounded estimate .... 54.65
Gamma Lower bound Upper bound
1.00 54.65 54.65
1.25 40.65 69.65
1.50 28.95 80.95
1.75 19.15 90.35
2.00 10.95 98.95
2.25 4.35 106.55
2.50 -1.95 113.35
Note: Gamma is Odds of Differential Assignment To
Treatment Due to Unobserved Factors
I have a spreadsheet to help with this. We need to identify the number of pairs with a clear winner, and the number of those “clear winner” pairs where the winner is the “treatment = 1” subject.
# A tibble: 2 × 6
matches treatment censored out_time subject propensity
<fct> <dbl> <dbl> <dbl> <chr> <dbl>
1 301 1 1 735 S001 0.516
2 301 0 0 572 S091 0.516
What if there is no censoring?
# A tibble: 4 × 6
matches treatment censored out_time subject propensity
<fct> <dbl> <dbl> <dbl> <chr> <dbl>
1 302 1 0 149 S005 0.487
2 302 0 0 144 S053 0.488
3 308 1 0 322 S082 0.524
4 308 0 0 369 S061 0.522
out_time?out_time?What if both subjects in the pair are censored?
# A tibble: 2 × 6
matches treatment censored out_time subject propensity
<fct> <dbl> <dbl> <dbl> <chr> <dbl>
1 310 1 1 75 S089 0.818
2 310 0 1 1095 S094 0.819
out_time?What if only the treated subject in the pair is censored?
# A tibble: 4 × 6
matches treatment censored out_time subject propensity
<fct> <dbl> <dbl> <dbl> <chr> <dbl>
1 301 1 1 735 S001 0.516
2 301 0 0 572 S091 0.516
3 307 1 1 460 S076 0.811
4 307 0 0 980 S273 0.813
out_time?out_time?What if only the control subject in the pair is censored?
# A tibble: 4 × 6
matches treatment censored out_time subject propensity
<fct> <dbl> <dbl> <dbl> <chr> <dbl>
1 305 1 0 595 S045 0.305
2 305 0 1 266 S350 0.306
3 337 1 0 194 S345 0.034
4 337 0 1 553 S197 0.033
out_time?out_time?out_time.out_time.
So, in total, we have 134 + 10 + 12 = 156 pairs with a clear winner. In 83 of those, the treated subject had the longer out_time.
toy exampleSection 18. What is a Sensitivity Analysis for Matched Samples?
We’ll study a formal sensitivity analysis approach for matched samples. Note well that this specific approach is appropriate only when we have
A formal sensitivity analysis asks: How would inferences about treatment effects be altered by hidden biases of various magnitudes? How large would these differences have to be to alter the qualitative conclusions of the study?
\(\Gamma\) is a measure of the degree of departure from a study that is free of hidden bias. We will specify different levels of hidden bias to see how large a \(\Gamma\) we can have while still retaining the fundamental conclusions of the matched analysis.
A sensitivity analysis will consider possible values of \(\Gamma\) and show how the inference for our outcomes might change under different levels of hidden bias, as indexed by \(\Gamma\).
rbounds packageSection 19 of the toy example provides a detailed example for a quantitative outcome, and notes that rbounds has additional functions to help evaluate sensitivity when we have a binary outcome.
Details in class.
In observational studies, investigators have no control over the treatment assignment. The treated and non-treated (that is, control) groups may have large differences on their observed covariates, and these differences can lead to biased estimates of treatment effects.
The propensity score, defined as the conditional probability of being treated given the covariates, can be used to balance the covariates in the two groups, and therefore reduce this bias.
Once estimated the propensity score can be used to reduce bias through matching, stratification (subclassification), regression adjustment, or some combination of all three.
Discussion of three main techniques (building on Rosenbaum and Rubin)
500 Class 7 | 2026-03-05 | https://thomaselove.github.io/500-2026/